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[57] ABSTRACT 

This invention relates to a method of allocating new 
telecommunications cells to one of a plurality of proces- 
sors, and to control acceptance of new calls when in 
overload. The method is based on measuring the real 
time occupancy of each processor periodically, and 
allocating new calls for the following period in such a 
way as to attempt to make each processor's occupancy 
approach the average occupancy of all the processors. 
An overload state is detected when the average occu- 
pancy of the processors in one period exceeds a prede- 
termined threshold, e.g., 90%. When this happens, load 
is shed in an amount calculated to restore the average 
occupancy to the threshold value over a number of 
periods of time. Advantageously, this method optimizes 
the call processing capacity of the processors. 

15 Claims, 2 Drawing Sheets 
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capacity of a plurality of processors to maximize the 
LOAD BALANCING AND OVERLOAD CONTROL total system capacity. 
IN A DISTRIBUTED PROCESSING 

TELECOMMUNICATIONS SYSTEM SOLUTION 

5 The above problem is solved and an advance is made 
TECHNICAL FIELD over the prior art in accordance with this invention 

This invention relates to work allocation to proces- whcrdn ' m m illustrative embodiment, the average real 
sorsin a distributed processing system. time work occupancy of each processor is measured 

periodically and based on this occupancy, the fraction 
PROBLEM 10 of new calls to be allocated to each processor during the 

In certain stored program controlled telecommunica- 116x1 1*™* 1 " m such a manner as to attempt to 

tions switching systems, such as AT&T's Autoplex tm "V»U*e occupancy of all the processors during that 
1000 system for serving cellular mobile radio telecom- penod - 10 ^ mustrative embodiment, each of the pro- 
munications traffic, a plurality of processors are avail- ,< cesso , ra measure their occupancy simultaneously and in 
able for performing the data processing operations nee- synchronism and are polled periodically by a call allo- 
essary to control each call in the system. Each of the < ? tl0 ° proces ! or ^ 0811 allocation processor adjusts 
plurality of processors is able to perform identical call of new calls to be allocated to each proces- 

processing functions required to serve such calls so that 801 dunng ™? next P^ 00 by "arcing that fraction for 
any new call can be served by any of these processors. 20 P rocessors * hose occupancy exceeds the average, and 

In such an arrangement, in order to maximize system mcreasm 8 the fraction forprocessors whose occupancy 
capacity and minimize call setup delays, it is important 1S „ leSS ^ ^ avera S e - The fraction S u of calls to be 
to have a quality method of allocating each newcall to f^**?™ 8 penod 1 to the rth iwocessor is equal to 
an appropriate processor. It is also important to have a s i/=Sw+(A-a,)G,, wherein So/ equals the fraction of 
quality method for responding to overload conditions, 25 ^ocat^ to that processor dunng period 0 

so that the telecommunications traffic handled when prev,ous T?*' t M the . avera 8 c . occ»Pancy of all 

subjected to overload is equal to or at least close to the P roc f* or ? «•*> P";™" 8 Pf nod > •/» occu- 
maximum amount of traffic that the system can handle °. f processor dunng the previous penod, 

when not in overload and that additional calls in over- Sj^jf,!! *™ i^**™* by Simula- 

load do not interfere with the processing of existing 30 * on experience. FoUowmg unplementation of 

calls or those new calls already accepted by the systenf * e ^ onthm . our mventon proves for a post solution 

The amount of processor time retired for each call ^IZ ^Z^^TT ^ ^ 

varies substantially depending in Ae case of cellular "T* P"* 5 " 8018 - Advantageously such an arrange- 

„„ t . / y „ ° , . ™r" ment makes it possible to divert traffic from heavily 

radio on the number of cell boundaries and switching . , . M . . . . " ' 

cx „ tm i i„ •„ .1... _ 4 • 8 35 loaded (though not overloaded) processors toward 

system boundaries that a mobile traverses in the course " ,• i,n„ iT~,j_j , * . . , iL 

of its call, and on the features invoked by the call. These ,T P"** 850 " " d tends to 

~ vu»«,™ H uw i u™Muu, tu 6 «ul iucsc \ oa £ evenly thus improving the capacity and resoon- 

11 ?i V* sivencssofthc system' Ad vLtageou^the respect 

S^o Hl": i n 1°^^ ncM * maximized because a balanced load minimizes 

allocated to a particular processor can be controUed mcan processing delays. A processor's occupancy can 
acutely .the amount of proc^ 40 be contributed to, in part, by non-call proceS loads 

a penod of minutes for processing these calls will still (c . g ., caU assignment base updates) which are 

vary by a substantial quantity. variable and unpredictable. This scheme automatically 

J? a 5"° r ^ raDgeme ? t . for * e Attt °P T lex 1000 accounts for such side loads by shifting the call Ioa4 

S ^T«^^ J* f™^ m 7^ ?."S U ' S * ^ « which 15 * c ^J 01 ** of load at time in any of 
No. 4,827,499, the fraction of new load which was alio- the processors, on the aggregate of processors, 
cated to each processor was determined initially by If the average occupancy of the processors exceeds a 
imtiakzation parameters that were subject to modifica- threshold T (Le., if A>T), or if the system was in the 
tion if any processor went into an overload condition. overload state during the previous interval (i.e., if Pq, 
Overall system overload (as opposed to individual pro- ^ the fraction of new calls accepted is less than 1), then a 
cessor overload) was recognized when the fraction of determination is made to allow a fraction Pi of offered 
individual processors that were overloaded exceeded a calls to be accepted, where Pi is the percentage previ- 
tnreshold; then the percentage of new calls that were ously allowed (P 0 ), plus a multiple of the difference 
rejected was gradually and linearly increased at a fixed between T, the level of average occupancy of the pro- 
rate as long as the system was in overload and decreased 55 cessors at the threshold of overload, and A, the average 
in the same way when the system went out of overload. measured occupancy. In one specific embodiment, 
Using this arrangement, individual processors close to 
overload tended to be allocated too much traffic while 

underloaded processors were allocated too little, and |~ G\ ~| 

the response to system overload was less than optimum. 60 ^i = ^Vj 1 + (7* - — J ( 

In view of the foregoing, a problem of the prior art is 
that in a distributed processing telecommunications where G% is a feedback stabilization factor. Simulations 
control system wherein calls may be allocated to differ- on the Autoplex 1000 system have indicated that a value 
ent processors, there is not a quality arrangement for of Ch=3 is appropriate. For example, if T=0.9, 
optimizing the traffic capacity of each processor and 65 A =0.95 and G2=3, and P 0 =l, signifying a sudden 
the performance of the system as a whole in overload. overload at the 0.95 occupancy load, where 0.9 repre- 
More generally, in distributed processing systems, there sents the threshold, then Pi =0.85, or 15 per cent of load 
is not a good system for equitably distributing the traffic is shed. This fraction is then broadcast from the allocat- 
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ing processor to all new call request generating systems of working call control processors in the system, is 

to request that they accept the specified fraction of new appropriate. The value of Gi must be changed if one or 

calls and reject or shed the remainder. Advantageously, more of the CCPs is temporarily out of service. The 

such an arrangement allows the system to go into a load period over which occupancy measurements are taken, 

shedding mode quickly when necessary, and to stop 5 and the call splits are adjusted, is also important; Simula- 

shedding load unnecessarily as soon as possible. tion studies for the Autoplex 1000 indicate that 10 sec- 

Accordingly, this invention is an arrangement for onds is a good choice for that system. The new alloca- 
distributing new load to one of a plurality of processors tion increases the fraction of new calls assigned to pro- 
of a distributed processing system by measuring the cessors whose occupancy was below the average and 
occupancy of each of the processors and adjusting the 10 decreases the fraction for processors whose occupancy 
Fraction of new load to be allocated to each processor to was above the average. Based on these new calculated 
attempt to equalize the occupancy of each of the pro- splits, a call allocation sequencer is regenerated to re- 
cessors * fleet the new splits; the call allocation sequencer is a 
BRIEF DESCRIPTION OF THE DRAWING for implementation of variable percentage round- 
„ 13 robin allocation. 

FIG. 1 is a block diagram of the control system of a Methods for implementing such variable round-robin 

pnor art cellular telecommunications system; dlocations „ wcll fa ^ ^ fc ^ 

J? 2 a °i 3 Pr0gram ? a »P«*fc implementation, given the splitting percent- 

? determimng the allocation ages , a deterministic CCP call allocation sequence is 

At* 1 f °!i detCr " 20 * enerated that approximates the sphtting Ventages, 

niimng how much load to shed in overload; and For exam le> tf a F £ C P's sphtting pircentoge is 25, one- 

m ™M rJj^na 3X31 ° f ™ ™^ quarter of 1 the entries in the assist seqLce wiSte 

SZSZL T 8 Pr<>CeSS m ^ t0 ** CCR ^ ^encfentries of a givenCCP 

p * are evenly distributed throughout the sequence (to the 

DETAILED DESCRIPTION 25 degree possible). This ensures that calls are distributed 

In the Autoplex 1000 system as described, for exam- ILjjTS ZJ™*^ ~ 

pie, mU.S.Pat.No.4,827;499,a P luraUtyofcallcontrol ttrS^T ^ " ™ V ^ Ctar,y ^ 

processors (CCPs) is used for serving calls. FIG. 1 is a 5S .£ th ^ uence detenmnes the accuracy with 

block diagram of the control system of the Autoplex 30 * 

1000 system, modified according to the present inven- ^ embodiment, a length of 64 was selected This was 

tion. Any new call coming into the system can be as- J"?* maximum of 6 proccs«)rs. More gener- 

signed to any of these CCPs (5,7). T^e assignment is f ° r ? P™*^* ^ ° f 10 n appears to * 

performed by one of these call control pSSi T*° P ?T ^ ^ l^P™* waIks 

called for convenience, an allocation/call control pro- 35 ~° p U ? h ^ a ? slgmn ? the next to the next 

cessor 5 (A/CCP). As shown in FIG. 3, each CCP 5,7 v m the sequence ( wra PP m 8 around when neces- 

measures its own average occupancy over a period, in Sa ^?' , . ' 

this case, ten seconds. FIG. 1 also shows a plurality of algorithm that generates the call assignment 

digital circuit switches (DCS) 9 for switching call con- ^ e ° c \ attemp . ts . for cach sequence element j to pick 

nections. The DCS generate new call requests. These 40 (0 f difference between 

are transmitted over cellular message switch 1, a ring V™*an <* entnes already allocated to CCP(i); in the 

message switch with a plurality of nodes 3 for commu- Vrr^ 1 "* mdth£can percent- 

nicating among processors and switches. The new call age CCP W' 

requests are transmitted to A/CCP 5 for allocation to 

itself or one Of the Other CCPs 7. The invention is im- 45 for each sequence clement j choose i to minimize 

plemented by program 6, part of which is described in y\/S)Su where: 

FIG. 2 and Which resides in A/CCP 5; and program 8, d(l is the proportion of sequence entries already given 

included within program 6, described in FIG. 3, which to CCP(Q in the first j-i sequence entries 

resides in each of the processors 5,7. Si/ is the call split for ccpq) 

Periodically, herein every ten seconds, the A/CCP 50 

polls each of the other CCPs and itself to receive the At each call load balancing interval, the splitting proba- 

measurement of occupancy (action block 101, FIG. 1) bilities are re-calculated and the call assignment se- 

and to start another period of occupancy measurement quence for the next interval is generated 

The A/CCP calculates the average occupancy (A) by Blocks 109, 111, and 113 relate to overload control 

averaging the individual occupancies a/ (action block 55 Test 109 checks whether the system is already in the 

1(G). For load balancing purposes, the A/CCP then call shedding mode or if the average occupancy exceeds 

calculates a new split of allocation of new calls for the a threshold T. The latter test is the test used for entering 

coining period according to the algorithm Si,=So/+- the call shedding mode wherein a fraction of new calk 

« « ! * W 7 erem Slu ^ re P resent fraction of is simply not accepted by the system. In this particular 

calls allocated to the l'th processor during the next 60 embodiment, calls are shed at random and no prefer- 

lnterval and the previous interval, respectively, A is the enceis made for either incoming land-originated mobile 

average occupancy of the processors as calculated in terminating calls or mobile originated calls It might 

action block 103, a/ is the occupancy reported by the i'th under some circumstances be desirable to shed more of 

processor, and Gi is a feedback gain parameter which is the mobile originated calls when call shedding is neces- 

tuned to yield the desired dynamic response. Gi is a 65 sary, since no resources have yet been used for such 

function of the number of call control processors. Simu- calls whereas land-originated calls have already had 

lation studies for the Autoplex 1000 system have indi- substantial resources used in setting up the connection 

cated that a value for G| = 1/n, where n is the number to a mobile switching office. However, the disadvan- 
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tage is that if this were done, then under overload mo- 1, the call message is dropped and A is decremented by 
bile units would be able to receive most calls but would 1; otherwise the call message is sent 
have a much lesser chance of being able to originate FIG. 3 shows the measurement process which is 
calls and this may be considered undesirable service to executed in each CCP7 and in A/CCP5. The length of 
the mobile units. 5 real time segments is accumulated throughout the per- 

If the result of test 109 is negative, then there is no iod of ten seconds (action block 201). The processor 
need to perform any call shedding and the actions of then receives a polling request from A/CCP5 (action 
this particular sequence is terminated (action block block 203). In response, the processor converts the 
115). If the result of test 109 is positive, then a new accumulated number into a percentage, representing 
calculation is made for the percent of calls to be shed 10 the processor occupancy, transmits this occupancy 
based on the present value of A. This calculation is measure to the A/CCP (action block 205) and resets the 
shown m action block 111, is based on the algorithm value of the real time segment accumulator to begin 

another cycle of measurement 

[ -i It is to be understood that the above description is 

l + (T- a)-j- I 15 only of one preferred embodiment of the invention. 

-J Numerous other arrangements may be devised by one 

, . 0 n , * . skilled in the art without departing from the spirit and 

wherein Pi, Po represent the new and previous percent- scope of the invention. The invention is thus limited 
age of new calls accepted (if the system is just entering only ^ defined m ^ claims . 
overload, then P 0 = 1), T is a predetermined threshold 20 ^ e claim . Ra "* * Wttuua * 

of average occupancy for invoking overload ^control, i. fa a distributed multiprocessing telecommunica- 

and G 2 is a gam factor which is tuned to yield the de- i . . r 

sired dynamic response. In the exemplary Lbodiment ^^T^TrT C ° mp T g * t^V^^' 
of theinvention based onsimulations^ lvalue selected ?* P roce f ,n 8 calls, a method of allocating 

for G 2 is 3, and the value selected for T is 0.9. In tins 25 IZ^e J™ f P * prOCeSSOrS COm ' 
calculation, Pi is truncated so that it can never exceed 1 . . 1 ps 01. 

and is never less than 0.05. The fraction Di of calls shed, r «*™& * measurement of real time work occu- 
l _ p ig. i» ™* pancy of each of said plurality of processors; and 

' responsive to said received measurements, adjusting 

30 an allocation of new calls to each processor to 
-j equalize the occupancies of each of said processors, 

l + (T-a)-j- J, 2. In a distributed multiprocessing telecommunica- 

tions control system comprising a plurality of proces- 
AvalueofG 2 abovelisdesirableinordertorespond 5015 ^ for P rocc f m 8 » method of aUocating 
promptly to sudden overloads. The value of A is limited 35 new . to ones of plurahty of processors corn- 
to 1 and if A is at that value, the actual overload may be pnsmg . . &tcps of: 

far greater than that implied by the difference between receiving a measurement of real time work occu- 
1 and T. Therefore, a sharp decrease in accepted load, panc ? of ^ of &aid P luraUtv of processors; 

made possible by ahigh value of G2, protects the system responsive to said received measurements, adjusting 
against excessively long message queues and processing 40 allocation of new calls to each processor to 

delays, which result in degraded performance in han- equalize the occupancies of each of said processors; 

dling both old and new calls. A high value of G 2 also detecting that an average of said measured occupan- 
leads to a more rapid recovery from overload controls. acs of said processors exceeds a predetermined 

It is possible that field experience will suggest the use of threshold; and 

a high value of G 2 for imposing overload controls, i.e., 45 shedding a fraction of new calls, said fraction deter- 
decreasing Pi whereas a lower value of G2 may be to chan « e m average occupancy of said 

appropriate for releasing overload controls, i.e., increas- processors to approximately equal said threshold, 

ing Pi. 3. The method of claim 2 wherein said shedding com- 

The value of P t is then broadcast to all the nodes P™* ^J^^g »"* shed fraction by a multiple of a 
which transmit call messages to the processors (action 50 difference between a predetermined threshold value 
block 113). These nodes will then delete a fraction md ^ avcra ge of said measured occupancies of said 
1-P, of new call messages and simply not forward Plurality of processors. 

these deleted messages to the A/CCP for allocation to 4 * The method of claim 2 wherein said shed fraction 
processors. is determined periodically by an algorithm: 

The deletion process can be carried out by the fol- 55 
lowing algorithm which is executed for each new call - 

A-i-Jb[i + a--4>-S-]. 

if a ° ~ ?) ' 60 wherein Dl represents said shed fraction for a next 
then drop the can message; P eriod ' p o represents a fraction of calls accepted during 
a = a - 1 a period immediately prior to said next period, T repre- 
ss* sents a predefined constant representing an average 
send the call message occupancy of said plurality of processors at a threshold 

■ . 65 of overload of said telecommunications control system, 

A represents average measured occupancy of said plu- 

Briefly, a parameter A, initially 0, is incremented by rality of processors during said prior period and G2 

the fraction of calls to be deleted. Whenever A exceeds represents a predetermined gain factor. 
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5. In a distributed multiprocessing telecommunica- 
tions control system comprising a plurality of proces- 
sors each for processing calls, a method of allocating 
new calls to ones of said plurality of processors com- 
prising the steps of; 

receiving a measurement of real time work occu- 
pancy of each of said plurality of processors; and 

responsive to said received measurements, adjusting 
an allocation of new calls to each processor to 
equalize the occupancies of each of said processors; 

wherein said adjusting is performed periodically and 
comprises adding to an allocated fraction for each 
processor for a prior period a predetermined multi- 
ple of a difference between an occupancy of said 
each processor during said prior period and an 
average occupancy of said plurality of processors 
during said prior period. 

6. The method of claim 5 wherein said allocated frac- 
tion for an i'th one of said plurality of processors is 
determined by an algorithm: 
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wherein Si/, So/ represent an allocated fraction for said 
i'th processor for a next and said prior period, respec- 
tively, A represents an average occupancy of said plu- 
rality of processors during said prior period, a/ repre- 
sents an occupancy of said i'th processor during said 
prior period, and Gi represents a predetermined gain 
factor. 

7. In a distributed multiprocessing telecommunica- 
tions control system comprising a plurality of proces- 
sors each for processing calls, a method of allocating 
new calls to ones of said plurality of processors com- 
prising the steps of: 

receiving a measurement of real time work occu- 
pancy of each pf said plurality of processors; and 

responsive to said received measurements, adjusting 
an allocation of new calls to each processor to m 
equalize the occupancies of each of said processors; 

wherein said occupancy of each of said processors is 
measured periodically over the same period, and 
said adjusting is performed at a beginning of each 
period. 45 

8. The method of claim 7 wherein each of said proces- 
sors is polled periodically to transmit said measurement. 

9. A method of assigning work to a plurality of pro- 
cessors comprising: 

determining a work occupancy for each of said pro- 50 
cessors; 

responsive to said occupancy determining, adjusting 
allocation fractions of new work for each of said 
processors toward work occupancy equalization 
among said processors; and 

thereafter allocating new work to said processors 
based on said adjusted allocation fractions. 

10. In a distributed multiprocessing system compris- 
ing a plurality of processors each for performing data 
processing operations in response to requests, a method 60 
of allocating new requests to ones of said plurality of 
processors comprising the steps of: 

measuring a work occupancy of each of said plurality 
of processors; 

responsive to said measuring, periodically adjusting 65 
allocation fractions of new work requests for each 
processor toward work occupancy equalization 
among said processors; and 



55 



thereafter, allocating new work to said processors 
based on said adjusted allocation fractions. 

11. The method of claim 10 wherein said adjusting 
comprises adding to an allocation fraction for each 
processor for a prior period a predetermined multiple of 
a difference between an occupancy of said each proces- 
sor during said prior period and an average occupancy 
of said plurality of processors during said prior period. 

12. In a distributed multiprocessing telecommunica- 
tions control system comprising a plurality of proces- 
sors each for processing calls, a method of allocating 
new calls to ones of said plurality of processors com- 
prising the steps of: 

measuring a work occupancy of each of said plurality 
of processors; 

periodically adjusting a fraction of new calls to be 
allocated to each processor in such manner that the 
occupancies of each of said plurality of processors 
are adjusted toward equalization; 

detecting that an average of said measured occupan- 
cies of said processors exceeds a predetermined 
threshold; and 

shedding a fraction of new calls, said fraction deter- 
mined to attempt to achieve an average occupancy 
of said processors approximately equal to said 
threshold; 

wherein said shedding comprises adjusting said shed 
fraction by a multiple of a difference between a 
predetermined threshold value and a measured 
average occupancy of said plurality of processors; 

wherein said shed fraction is determined periodically 
by an algorithm: 

d x « i - + ( r-x)-2_], 

wherein D\ represents said shed fraction for a next 
period, Po represents a fraction of calls accepted 
during a period immediately prior to said next 
period, T represents a predefined constant repre- 
senting an average occupancy of said plurality of 
processors at a threshold of overload of said tele- 
communications control system, A represents aver- 
age measured occupancy of said plurality of pro- 
cessors during said prior period and Gi represents 
a predetermined gain factor; 

wherein said adjusting comprises adding to an allo- 
cated fraction for each processor for a prior period 
a predetermined multiple of a difference between 
an occupancy of said each processor during said 
prior period and an average occupancy of said 
plurality of processors during said prior period; 

wherein said allocated fraction for an i'th one of said 
plurality of processors is calculated by a formula: 

wherein Su, So/ represent an allocated fraction for 
said i'th processor for a next and said prior period, 
respectively, A represents an average occupancy 
of said plurality of processors during said prior 
period, a/ represents an occupancy of said i'th pro- 
cessor during said prior period, and Gi represents a 
predetermined gain factor; and 
wherein said occupancy of each of said processors is 
measured periodically over the same period, and 
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said adjusting is performed at a beginning of each 
period. 

13. In a distributed multiprocessing telecommunica- 
tions control system, means for allocating new calls to 
ones of a plurality of call control processors, compris- 
ing: 

a plurality of call control processors, each operative 
under the control of a first program, for measuring 
its own real time work occupancy; and 

processor means, operative under the control of a 
second program and responsive to said measuring 
real time work occupancy of each of said call con- 
trol processors, for adjusting an allocation of new 
calls to each of said call control processors to 



10 



IS 



equalize said real time work occupancies of each of 
said call control processors. 

14. The means for allocating new calls of claim 13 
wherein said call control processors are further opera- 
tive under the control of said first program for measur- 
ing said occupancies periodically and for reporting said 
measurements to said processor means periodically. 

15. The means for allocating new calls of claim 13 
wherein said processor means is further operative under 
the control of said second program to determine 
whether an average of said measured occupancies ex- 
ceeds a predetermined threshold, and, responsive to 
said determination, for controlling the shedding of a 
fraction of new calls, said fraction determined to change 
an average occupancy of said call control processors to 
approximately equal said threshold. 

***** 
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